High performance computing often requires access to data and files stored on relatively slow non-volatile stable storage devices, such as hard drives. Overall computing performance may be improved by improving file access performance. Caches that use high-speed volatile storage placed close to the data write source may often improve high performance file and data access. Data caches may improve performance by reducing the time and overhead required for writes to complete. However, data caches that use volatile unstable storage may lose data if there are power outages or other failure events. Accordingly, data caches may be arranged to regularly flush data from volatile unstable storage to non-volatile stable storage.
Often, data caching must be suspended while flushing file data from high-speed unstable storage to slower stable storage. Otherwise, as data may be moved from unstable storage to stable storage new data may be written into the data cache at the same time data may be in the process of being moved to stable storage. This may make it difficult or impossible to avoid data and file system inconsistency.
Also, for computing systems with large and/or distributed data caching facilities it may take a relatively long time for file data caches to be flushed from unstable storage to stable storage. Suspending data caching for a long period of time may significantly impact the performance of computing systems. Likewise, in some distributed file systems enabling the archival of data that is stable and consistent may be difficult during file system operations. Thus, it is with respect to these considerations and others that the invention has been made.